Storage system including a switch

ABSTRACT

A disk storage system containing a storage device having a record medium for holding the data, a plurality of storage sub-systems having a controller for controlling the storage device, a first interface node coupled to a computer using the data stored in the plurality of storage sub-systems, a plurality of second interface nodes connected to the storage sub-systems, a switch connecting to a first interface node and a plurality of second interface nodes to perform frame transfer therebetween based on node address information added to the frame. The first interface node has a configuration table to store structural information for the memory storage system and in response to the frame sent from the computer, analyzes the applicable frame, converts information relating to the transfer destination of that frame based on structural information held in the configuration table, and transfers that frame to the switch.

This is a continuation application of U.S. Ser. No. 09/468,327, filed onDec. 21, 1999.

BACKGROUND OF THE INVENTION

This invention relates to a disk control system for controlling aplurality of disk devices and relates in particular to a method forimproving the high speed operation of the disk control system, achievinga lower cost and improving the cost performance.

A diskarray system for controlling a plurality of disk devices isutilized as a storage system in computers. A diskarray system is forinstance disclosed in “A Case for Redundant Arrays of Inexpensive Disks(RAID)”; In Proc. ACM SIGMOD, June 1988 (Issued by Cal. State Univ.Berkeley). This diskarray operates a plurality of disk systems inparallel and is a technique that achieves high speed operation comparedto storage systems utilizing disks as single devices.

A method using the fabric of a fiber channel is a technique for mutuallyconnecting a plurality of hosts with a plurality of diskarray systems. Acomputer system using this technique is disclosed for instance in“Serial SCSI Finally Arrives on the market” of Nikkei Electronics, P.79, Jul. 3, 1995 (No. 639) as shown in FIG. 3. In the computer systemdisclosed here, a plurality of host computers (hereafter simply calledhosts) and a plurality of diskarray systems are respectively connectedto a fabric device by way of fiber channels. The fabric device is aswitch for the fiber channels and performs transfer path connectionsbetween the desired devices. The fabric device is transparent to (orpasses) “frame” transfers which are packets on the fiber channel. Thehost and diskarray system communicate between two points withoutrecognizing he fabric device.

SUMMARY OF THE INVENTION

In diskarray systems of the conventional art, when the number of diskdevices were increased in order to increase the storage capacity andachieving a controller having high performance matching the number ofdisk units was attempted, the internal controller buses were found tohave only limited performance and likewise, the processor performingtransfer control was also found to have only limited performance. Inorder to deal with these problems, the internal buses were expanded andthe number of processors was increased. However, attempting to solve theproblem in this manner made the controller structure more complex due tothe control required for a greater number of buses and caused increasedoverhead and complicated software control due to non-exclusive controlof data shared between processors, etc. The rise in cost consequentlybecame extremely high and performance reached its limits so that costperformance was unsatisfactory. Though the cost for this kind could bejustified in terms of performance in a large scale system, in systemsnot or such a large scale the cost did not match performance,expandability was limited and the development period and developmentcosts increased

The overall system storage capacity and performance can be increased byconnecting a plurality of diskarray systems in parallel with a fabricdevice. However, in this method, there is absolutely no connectionbetween the diskarray systems, and access concentrated on a particulardiskarray system cannot be distributed among the other devices so thathigh performance cannot be achieved in actual operation. Also, thecapacity of a logical disk device (hereafter logic unit) as seen fromthe host is limited to the capacity of one diskarray system so that ahigh capacity logic unit cannot be achieved.

In an attempt to improve diskarray system reliability, a diskarraysystem can be comprised of a mirror structure where, in two diskarraysystems, the host unit has a mirroring function. However, this methodrequires overhead due to control required of the mirroring by the hostand also has the problem that performance is limited. This method alsoincreases the load that the system administrator must supervise sincemany diskarray systems are present inside the system. The maintenancecosts thus increase since a large number of maintenance personnel mustbe hired and maintenance fees must be paid for each unit. The pluralityof diskarray systems and fabric devices are further all autonomousdevices so that the settings must be made by different methods accordingto the respective device, creating the problem that operating costsincrease along with a large increase in operating time and systemadministrator training time, etc.

In order to resolve these problems with the related art, this inventionhas the object of providing a disk storage system capable of beingstructured according to the scale and requirements of the computersystem, and a disk storage system that responds easily to needs for highreliability and future expansion.

The disk storage system of this invention contains a storage devicehaving a record medium for holding the data, a plurality of storagesub-systems having a controller for controlling the storage device, afirst interface node coupled to a computer using the data stored in theplurality of storage sub-systems, a plurality of second interface nodesconnected to any or one of the storage sub-systems, a switch connectingbetween a first interface node and a plurality of second interface nodesto perform frame transfer between a first interface node and a pluralityof second interface nodes based on node address information added to theframe.

The first interface node preferably has a configuration table to storestructural information for the memory storage system and a processingunit to analyze the applicable frame in response to the frame sent fromthe computer, converts information relating to the transfer destinationof that frame based on structural information held in the configurationtable, and transfers that frame to the switch. Further, whentransmitting a frame, the first interface node adds the node addressinformation about the node that must receive the frame, to that frame. Asecond interface node then removes the node address information from theframe that was received, recreates the frame and transfers that frame tothe desired storage sub-system.

In the embodiment of this invention, the disk storage system has amanaging processor connecting to the switch. The managing processor setsthe structural information in the configuration table of each nodeaccording to the operator's instructions. Information for limitingaccess from the computer is contained in this structural information.

In another embodiment of this invention, the first interface nodereplies to the command frame sent from the computer instructing thewriting of data, makes copies of that command frame and the followingdata frames, adds different nodes address information to each frame sothe received frame and the copied command frames will be sent to thedifferent respective nodes and sends these frames to the switch.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing the structure of the computer systemof the first embodiment of this invention.

FIG. 2 is block diagram of the diskarray subset of the first embodiment.

FIG. 3 is block diagram of diskarray switch of the first embodiment.

FIG. 4 is a block diagram of the crossbar switch of the diskarray switchof the first embodiment.

FIG. 5 is block diagram of the host I/F node for he diskarray switch ofthe first embodiment.

FIG. 6A is sample diskarray system configuration table

FIG. 6B is sample diskarray system configuration table.

FIG. 7 is a block diagram of the frame of the fiber channel.

FIG. 8 is a block diagram of the frame header of the fiber channel.

FIG. 9 is a block diagram of the frame payload of the fiber channel.

FIG. 10 is a model view showing the sequence of frames sent by way ofthe fiber channel during read operation from the host

FIG. 11 is a model view showing the interactive relationship of thehost-LU, the LU for each diskarray subset, as well as each diskarrayunit.

FIG. 12 is a block diagram of the S packet.

FIGS. 13A through 13C are flowcharts of the processing in the host I/Fnode during write processing.

FIG. 14 is a block diagram showing a plurality of diskarray switches ina cluster-connected diskarray system.

FIG. 15 is a block diagram of the computer system of the secondembodiment of this invention.

FIG. 16 is a block diagram of the diskarray switch IC of the fourthembodiment of this invention

FIG. 17 is a block diagram of the computer system of the fifthembodiment of this invention.

FIG. 18 is a screen configuration view showing a typical display of thelogic connection structure.

FIG. 19 is a model diagram showing the frame sequence in the sixthembodiment of this invention.

FIGS. 20A through 20D are flowcharts showing the processing on the hostI/F node during the mirroring write processing in the sixth embodimentof this invention.

FIG. 21 is an address spatial diagram of the diskarray system for theseventh embodiment of this invention.

FIG. 22 is a flowchart showing the processing in the host I/F node ofthe seventh embodiment of this invention.

FIG. 23 is a block diagram of the disaster recovery system of the eightembodiment of this invention

FIG. 24 is a descriptive view of the alternative path setup.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

(First Embodiment)

FIG. 1 is a block diagram showing the structure of the computer systemof the first embodiment of this invention. In the figure, referencenumeral 1 denotes a diskarray system, and 30 is the (host) computerconnected to the diskarray system. The diskarray system 1 contains adiskarray subset 10, a diskarray switch 20 and a diskarray systemconfiguration manager 70 for handling the configuration of the overalldiskarray system. The diskarray system 1 further has a communicationinterface (communication I/F) 80 between the diskarray switch 20 and thediskarray system configuration manager 70, and also between thediskarray subset 10 and the diskarray system configuration manager 70. Ahost 30 and the diskarray system 1 are connected by a host interface(host I/F)) 31. The host I/F 31 is connected to the diskarray switches20 of the diskarray system 1. The diskarray switch 20 and the diskarraysubset 10 inside the diskarray system 1 are connected by the diskarrayinterface (diskarray I/F 21).

The hosts 30 and the diskarray subsets 10 are shown as four units eachhowever this number is optional and is not limited. The hosts 30 and thediskarray subsets 10 may also be provided in different numbers of units.The diskarray switches 20 in this embodiment are duplexed as shown inthe drawing. Each host 30 and each diskarray subset 10 are connected toboth of the duplexed diskarray switches 20 by the respective host I/F 31and a diskarray I/F 21. Thus even if one of the diskarray switches 20,the host I/F 31 or the diskarray I/F 21 is broken, the other diskarrayswitches 20, the host I/F 31 or the diskarray I/F 21 can be utilized toallow access from the host 30 to the diskarray system 1, and a highamount of usage can be achieved. However, this kind of duplication orduplexing is not always necessary and is selectable according to thelevel of reliability required by the system.

FIG. 2 is block diagram of a diskarray subset 10 of the firstembodiment. The reference numeral 101 denotes the host adapter forinterpreting the commands from the host system (host 10), executing thecache hit-miss decision and controlling the data transfer between thehost system and the cache. The reference numeral 102 denotes the cachememory/shared memory that comprises the cache memory for performing highspeed disk data access and a shared memory for storing data shared bythe host adapters 101 and the lower adapters 103. The reference numeral104 denotes a plurality of disk units stored inside the diskarray subset10. Reference numeral 103 is the lower adapter for controlling a diskunit 104 and controlling the transfer of data between the disk unit 104and the caches. Reference numeral 106 is the diskarray subsetconfiguration manager to perform communications between the diskarraysystem configuration manager 70 and the overall diskarray system 1, andalso manage the structural parameter settings and reporting of troubleinformation, etc. The host adapter 101, the cache memory/shared memory102, and the lower adapter 103 are respectively duplexed here. Thereason for duplexing is to attain a high degree of utilization, just thesame as with the diskarray switch 20 and is not always required. Eachdisk unit 104 is also controllable from any of the duplexed loweradapters 103. In this embodiment, the cache and shared memories jointlyutilize the same memory means in view of the need of low costs howeverthe caches and shared memories can of course be isolated from eachother.

The host adapter 101 comprises an host MPU1010 to execute control of theadapter 101, an host system or in other words a diskarray I/F controller1011 to control the diskarray switches 20 and the connecting I/F whichis the diskarray I/F 21, and an host bus 1012 to perform sircommunications and data transfer between the cache memory/shared memory102 and host MPU1010 and the diskarray I/F controller 1011. The figureshows one diskarray I/F controller 1011 for each host adapter 101however a plurality of diskarray I/F controllers 1011 can also beprovided for each one host adapter.

The lower adapter 103 contains a lower MPU103 to execute control of thelower adapter 103, a disk I/F controller 1031 to control the disk 104and interface which is the disk I/F, and a lower bus 1032 to performcommunications and data transfer between the cache memory/shared memory102 and host MPU1030 and the diskarray I/F controller 1031. The figureshows four diskarray I/F controllers 1031 for each lower adapter 103however the number of diskarray I/F controllers is optional and can bechanged according to the diskarray configuration and the number of disksthat are connected.

FIG. 3 is block diagram of the diskarray switch 20 of the firstembodiment. The diskarray switch 20 contains a Managing Processor (MP)which is a processor for performing management and control of the entirediskarray switch, a crossbar switch 201 for comprising n×n mutual switchpaths, a diskarray I/F node 202 formed for each diskarray I/F 21, a hostI/F node 203 formed for each host I/F 31, and a communication controller204 for performing communications with the diskarray systemconfiguration manager 70. The reference numeral 2020 denotes a path forconnecting the diskarray I/F node 202 with the crossbar switch 201, apath 2030 connects the host I/F node 203 and the crossbar switch 201, apath 2040 connects with the other diskarray switch 20 and other I/F forforming clusters, a path 2050 connects the MP200 with a crossbar switch201

FIG. 4 is a block diagram showing the structure of the crossbar switch201. A port 2010 is a switching part (SWP) for connecting the paths2020, 2030, 2050 and cluster I/F 2040 to the crossbar switch 201. Theswitching ports 2010 all have the same structure and perform switchingcontrol of the transfer paths to other SWP from a particular SWP Thefigure shows on SWP however identical transfer paths exist between allthe SWP.

FIG. 5 is a block diagram showing the structure of the host I/F node203. In this embodiment, use of a fiber channel is assumed for both thediskarray I/F 21 and the host I/F 31 in order to provide a specificdescription. The host I/F 31 and the diskarray I/F 21 can of course beimplemented with interfaces other than fiber channels. By utilizing anidentical interface, the host I/F node 203 and the diskarray I/F node202 can both have the same structure. In this embodiment, the diskarrayI/F node 202 has the same structure as the host I/F node 203 as shown inthe figure. Hereafter, the host I/F node 203 will be described by usingan example. A Searching Processor (SP) searches for what frame toconnect the fiber channel frame (hereafter simply called frame) to, anInterface Controller (IC) 2023 transmits and receives the frames withthe host 30 (the diskarray subset 10 when using the diskarray I/F node202), a Switching Controller (SC) 2022 performs conversion based onresults found by the SP2021 for frames received by the IC2023, aSwitching Packet Generator (SPG) 2024 packetizes the frame converted bythe SC2021 into a configuration that can pass the crossbar switch 201 totransfer to other nodes, a Frame Buffer (FB) 2025 temporarily stores thereceived frame, an Exchange Table (ET) 2026 supervises use of exchangenumbers for identifying a plurality of frame strings corresponding to adisk access request command (hereafter simply called command) from onehost, and a Diskarray Configuration Table (DCT) 2027 stores structuralinformation for a plurality of diskarray subsets 10.

Each structural section of the diskarray switch 20 are preferably allcomprised of hardware logic from the viewpoint of performance. However,program control utilizing general purpose processors is allowable forthe SP2021 and the SC2022 functions if the specified performance can beachieved.

Each diskarray subset 10 has disk units 104 as one or a plurality oflogical disk units. These logical disk units are referred to as LogicalUnits (LU) The LU need not correspond in a ratio of one to one, to thelogical disk units 104 and one disk unit 104 can be comprised of aplurality of LU or one LU can comprise a plurality of disk units 104 OneLU is recognized as one disk device as seen externally of the diskarrayunit 10. In this embodiment, a logical LU is comprised further by adiskarray switch 20 and the host 30 functions to access this LU. Inthese specifications, when one LU is recognized as one LU by the host30, then the LU is called independent LU (ILU) and when a plurality ofLUs are recognized as one LU by the host 30, then the one LU recognizedby the host 30 is called combined LU (CLU).

FIG. 11 shows the address spatial relation for each level when onecombined LU (CLU) is comprised of four LUs of four diskarray subsets. Inthe figure, the numeral 1000 indicates an LU address space for onecombined LU (CLU) of the diskarray system 1 as seen from the host “#2”,the numeral 1100 is an LU address space for the diskarray subset 10, thenumeral 1200 indicates an address space for the disk unit 104 (Here,shown only for the diskarray subset #0.) The LU for each diskarraysubset 10 is comprised as a RAID 5 (Redundant Arrays of InexpensiveDisks Level 5) type diskarray, by four disk units 104. Each diskarraysubset 10 has an LU with respective capacities of n0, n1, n2, n3. Eachdiskarray switch 20 combines the address spaces held by these four LU toobtain a combined capacity (n0+n1+n2+n3) and achieve a combined LU (orCLU) recognized from the host 30.

In this embodiment, when for instance the host #2 is accessing theregion A1001, an access request is made specifying the region A1001, andthis access request is converted by the diskarray switch 20 into arequest for accessing the region A′ 1101 of the LU of the diskarraysubset #0 and this request then sent to the diskarray subset #0. Thisdiskarray subset #0 then performs access and mapping of the region A′1101 onto of the region A″ 1201 on the disk unit 104. The mappingbetween the address space 1000 and the address space 1100 ES based onstructural information held in the DCT207 in the diskarray switch 20.The details of this processing are related later on. The mappingperformed in the diskarray subset is a technical method already wellknown in the prior art so a detailed explanation is omitted here.

In this embodiment, the DCT207 contains a Diskarray System ConfigurationTable and Diskarray Subset Configuration Tables. The structure of theDiskarray System Configuration Table is shown in FIG. 6A and thestructure of the Diskarray Subset Configuration Tables are shown in FIG.6B.

As shown in FIG. 6A, the Diskarray System Configuration Table 20270 hasa Host-LU Configuration Table 20271 holding information showing thestructure of the host-LU, and a Diskarray I/F Node Configuration Table20272 showing the related connections of the diskarray subset 10 and thediskarray I/F node 202 of the diskarray switch 20.

The Host-LU Configuration Table 20271 has LU information (LU Info.)relating to the condition and Host LU of the diskarray subset 10 LU,which is information showing the LU type, CLU class, CLU stripe size andHost-LU indicating the affiliation of the LU and the Host-LU No. whichis a number for identifying that LU. The LU Type in the table isinformation on the LU type showing that the Host-LU is a CLU or one LU.The CLU class is information showing the class is any one of “Joined”,“Mirrored” or “Striped” when the LU type of this Host-LU is shown to bea CLU. Here, “Joined” indicates as shown in FIG. 11 the CLU is one largememory space consisting of a group of LU connected together. As relatedlater in the sixth embodiment, “Mirrored” indicated two LU achieved by aduplexed LU. As related later on in the seventh embodiment, “Striped”indicates an LU stored with data distributed into a plurality of theseLU. When the CLU Stripe Size is shown by ‘Striped’ for the CLU class,then the striping size (A block size showing the units the data isdistributed in.) is indicated. The status shown in the Condition box isone of four types consisting of “Normal”, “Warning”, Fault and “NotDefined”. Of these types, “Normal” indicates the Host-LU status iscorrect “Warning” indicates contraction is being performed for reasonssuch as problems occurring in a disk unit corresponding to an LUcomprising this Host-LU. “Fault” indicates that this Host-LU cannot beoperated due to a problem in the diskarray subset 10. The “Not Defined”type indicates the Host-LU is not defined for the corresponding Host-LUNo. The LU Info contains information specifying the diskarray subset 10affiliated with that LU, the LUN inside the diskarray subset, as well asinformation showing the size for LU that comprise this Host-LU. When theHost-LU is an ILU, then information for the sole LU is registered. Whenthe Host-LU is a CLU, then information relating to all the respective LUcomprising that CLU are registered. In the figure for instance, aHost-LU with a Host-LU No. of “0” is a CLU comprised from four LU thatare LUN “0” of the diskarray subset “#0”, LUN “0” of the diskarraysubset “#1”, LUN “0” of the diskarray subset “#2”, and LUN “0” of thediskarray subset “#3”. As can be seen in the table, this CLU is in the“Joined” CLU class.

The diskarray I/F node configuration table 20272 contains information onwhat diskarray I/F node 202 of diskarray switch 20 is connected to eachport of the diskarray subset 10 connected to the diskarray I/F 21. Morespecifically, this table holds the Subset NO. specifying the diskarraysubset 10, the Subset Port No. specifying the port, the Switch Nospecifying the diskarray switch 20 connected to that port, and an I/FNode No., specifying the diskarray I/F node 202 of the diskarray switch20. When the diskarray subset 10 has a plurality of ports, informationis set for each of those ports.

As shown in FIG. 6B the diskarray subset configuration table has aplurality of tables 202720 through 202723 corresponding to each of thediskarray subsets 10. These tables include the RAID Group ConfigurationTable 202730 holding information showing the structure of the RAID Groupinside the diskarray subset 10, and the LU Configuration Table 202740holding information showing the structure of the LU inside the diskarraysubset 10,

The RAID Group Configuration Table 202730 has a Group No. showing thenumber added to the RAID Group, a level showing the RAID Level of thatRAID Group, and Disks Pith information showing the number of diskcomprising that RAID Group. When that RAID Group is comprised ofstriping such as for RAID Level 0, 5, then information showing thatStripe Size is included. As shown for instance, in the figure in thetable, a RAID Group “0” is a RAID Group comprised of four disk units.The RAID Level is 5 and the Stripe Size is 50.

The LU Configuration Table 202740 has an LU No. showing the number (LUN)added to the LU, a RAID Group showing how that LU is configured in theRAID Group, a Condition showing the status of the LU, a Size showing thesize (Capacity) of that LU, a Port showing what ports of the diskarraysubset 10 are capable of providing access, and also an Alt. Port showingport that can be used as alternates for that Port No. The status showingthe condition are of four types just as with the Host-LU and comprise“Normal”, “Warning” “Fault” and “Not Defined”. The port specified byinformation set in the Alt. Port is utilized when a problem occurs inthe port specified with information set in the Port (item) however canalso be used just for accessing the same LU from a plurality of ports.

FIG. 7 is a diagram of the frame for the fiber channel. A frame 40 ofthe fiber channel has an SOF (Start Of Frame) showing the beginningportion of the frame, a frame header 401, a frame payload 402 which is asegment storing data for transfer, a CRC (Cyclic Redundancy Check) 403which is a bit error detection code, and a EOF (End Of Frame) showingthe end of the frame. The frame header 401 has the structure shown inFIG. 8. The ID of the frame transfer originator (S_ID), the ID for theframe transfer destination (D_ID), Exchange IDs respectively specifiedby the Exchange Originator and the Exchange Responder (OX_ID, RX_ID),and the Sequence ID for specifying the frame group within the exchange(SEQ_ID) are all stored in the frame header 401. In this embodiment, theID assigned as S_ID to the host 30 in the frame issued from the host 30are also used as the ID assigned to the port of the diskarray switch 20as the D_ID. One pair of Exchange ID (OX_ID, RX_ID) are assigned for onehost command. When a plurality of data frames must be issued for thesame Exchange, then an identical SEQ_ID is assigned to all of these dataframes, and each one is identified as Sequence Count (SEQ_CNT). TheFrame Payload 402 has a maximum length of 2112 byte and the contentsstored in each type frame are different. In the case for instance ofFCP_CMD frame related later on, the Logical Unit Number (LUN) of theSCSI and the Command Description Block (CDB) are stored as shown in FIG.9. The CDB contains the command bytes required to access the disk(diskarray), the transfer start logic address (LBA) and the transferlength (LEN).

The operation of the disk address system of this embodiment is describednext.

In order to use the diskarray system, the setting of structuralinformation of the diskarray subset 10 must be made for the diskarrayswitch 20. The system administrator can acquired structural setupinformation for the diskarray switch 20 and the diskarray subset 10 froma management console 5 by way of the diskarray configuration manager 70.The administrator can make different kinds of required entries of setupinformation such as logic unit structural setup for the desired systemstructure, RAID level settings, alternative path settings for use whentrouble occurs. The diskarray configuration manager (means) 70 canreceive that setting information, and transfer that setting informationto the each diskarray subset 10 and diskarray switch 20. The entry ofsetup information on the management console 5 is described separately inthe fifth embodiment.

In the diskarray switch 20, the communications controller 204 acquiresshe setup information and sets he structural information such as theaddress space information for each of the diskarray subsets 10 by meansof the MP200. The MP200 distributes the structural information of hediskarray subset 10 to the each of the host I/F nodes 203 and thediskarray I/F nodes 202 by flay of the crossbar switch 201. When thenodes 202 and 103 receive this information, the SP2021 stored thisstructural information in the DCT2027. In the diskarray subset 10, thediskarray subset configuration manager (means) 106 acquires the setupinformation and stores it in the shared memory 102. The host MPU1010 andthe lower MPU 1030 refer to this setup information in the shared memory102 and perform configuration management.

The operation when the read command is issued is described next for thediskarray system 1 with a host “#2”. FIG. 10 is a model view showing thesequence of frames sent by way of the fiber channel during readoperation from the host FIGS. 13A through 13C are flowcharts of theprocessing in the host I/F node 203 during write processing In thefollowing description, it is assumed the host “#2” is accessing thestorage area A1001 in FIG. 11. The act al storage area A″ correspondingto the storage area A1001 is present in the address space of the diskunit #2 comprising the LU for LUN=0 of the diskarray subset “#1”. In thedefinition of the LU comprising the address space 1000, in theConfiguration Table 20271, the LU Type is defined as “CLU” and the CLUClass is defined as “Joined”.

During reading of data, the host 30 issues a command frame “FCP_CMD”stored with the read command, to the diskarray switch 20 (arrow (a) inFIG. 10) The host I/F node “#2” of the diskarray switch 20 receives thecommand frame “FCPCMD” (step 20001) by way of the host I/F 31 from theIC2023. The IC2023 transfers the command frame to the SC2022 The SC2022temporarily stores the received command frame in the Frame Buffer (FB)2025. At that time, the SC2022 calculates the CRC of the command frameand inspect the received information to determine if it is correct. Ifan error is found in the CRC inspection, the SC2022 reports the error tothe IC2023. When the IC2023 the error report from the SC2022, a reportof the CRC error is made to the host 30 by way of the host I/F 31 (step20002).

When the CRC inspection shows that the information is correct, theSC2022 reads the frame held in the FB202, recognizes this frame as thecommand frame, and analyzes the Frame Header 401 (step 20003). TheSC2022 then instructs the SP2021 and registers the Exchange informationsuch as S_ID, D_ID, OX_ID in the ET2026 (step 20004). Next, the SC2022analyzes the frame payload 402 and acquires the LUN and CDB specified bythe host 30 (step 20005). The SC2021 searches the DCT2020 at theinstruction of the SC2022 and acquires the structural information of thediskarray subset 10. More specifically, the SC2021 searches the host-LUconfiguration table 20271 and finds information having a host-LU no.matching the LUN stored in the frame payload 402 that was received. TheSC2021 recognizes the structure of the Host-LU from the information setin the LU Type, and CLU class, and based on the information held in theLU Info., identifies the disk subset 10 that must be accessed and itsLUN in the LU as well as the LBA in the LU Next, the SC2021 refers tothe LU configuration table 202740 of the Diskarray Subset ConfigurationTable 202720 and confirms the connection port for the destinationdiskarray subset 10, and acquires from the Diskarray I/F NodeConfiguration Table 20272, the node No. of the diskarray I/F node 202connected to that port. The SC2021 in this way acquires the conversioninformation such as the No. LUN, LBA for recognizing the diskarraysubset 10 and reports this information to the SC2022 (step 20006). Next,using the acquired conversion information, the SC2022 converts the LBAfrom the LUN and CDB of the frame payload 402. Also, the D_ID of theframe header 401 is converted to the D_ID of the host I/F controller1011 of the diskarray subset 10. The S_ID is not rewritten at this point(step 20007). The SC2022 transfers the converted command frame and thediskarray I/F node o. connected to the corresponding diskarray subset10, to the SPG2024. The SPG2024 generates a packet added with a simpleexpansion header 601 such as shown in FIG. 12 for the converted commandthat was received. This packet is called the Switching Packet (S Packet)60. The expansion header 601 of this S Packet 60 contains an addedtransfer originator (white node) No., a transfer responder node No. anda transfer length. The SPG2024 send the generated S Packet 60 to thecrossbar switch 201 (step 20008).

The crossbar switch 201 receives the S Packet 60 from the SWP2010connected to the host T/F node “#2”. The SWP2010 refers to the expansionheader 601 of the S Packet 60, establishes a path for carrying outswitch control for the SWP connecting with the transfer responder node,and transfers the S Packet 60 to the transfer responder of the diskarrayI/F node 202 (Here, the diskarray I/F node “#0”). The SWP2010establishes a path whenever the S Packet 60 is received and releasesthat path when transfer of the S Packet 60 is finished. In the diskarrayI/F node “#0”, the SPG2024 receives the S Packet 60, removes theexpansion header 601 and delivers the command frame portion to theSC2022. The SC2022 writes its own ID in the S_ID of the frame header ofthe command frame that was accepted. Next, the SC2022 instructs theSP2021 to register the Exchange information such as the S_ID, D_ID,OX_ID, of the command frame as well as the frame transfer originatorhost I/F node No. into the ET2026, and transfers this command frame tothe IC2023. The IC2023 complies with instructions of the frame header401 and transfers the command frame (arrow (b) of FIG. 10) to theconnected diskarray subset 10 (Here, the diskarray subset “#0”).

The diskarray subset “#0” receives the command frame “FCP_CMD” afterconversion, in the diskarray I/F controller 1011. The host MPU1010acquires the LUN and CDB stored in the frame payload 402 of the commandframe and recognizes that the LEN length data from the LBA of thespecified logical unit is the read command. The host MPU1010 refers tothe cache management information stored in the cache/shared memory 102and performs cache miss-hit/hit identification If a hit then the data istransferred from the cache 102. It a miss then reading of data from thedisk unit is necessary so that address conversion is implemented basedon the structure of RAID 5 and a cache space is secured. Processinginformation required for read processing from the disk unit 2 isgenerated, and processing information for continued processing in thelower MPU 1030 is stored in the cache/shared memory 102. The lower MPU1030 starts processing when the processing information is stored in thecache/shared memory 102. The lower MPU 1030 specifies an appropriatedisk I/F controller 1031 and generates a read command to the disk unit2, and issued a command to the disk I/F controller 1031. The disk I/Fcontroller 1031 stored the data read from the disk unit 2 in the addressspecified by the cache/shared memory 102 and issues a completion reportto the lower MPU 1030. The lower MPU 1030 stores the processingcompletion report in the cache/shared memory 102 for reporting to thehost MPU1010 that processing was completed correctly. The host MPU1010restarts the processing when the processing completion report is storedin the cache/shared memory 102 and reports that read data setup iscomplete to the diskarray I/F controller 1011. The diskarray I/Fcontroller 1011 issues a “FCP_XFER_RDY” which is a data transfer setupcompletion frame on the fiber channel for the applicable diskarray I/Fnode “#0” of the diskarray switch 20 (arrow (c) of FIG. 10). In thedisks ray I/F node “#0”, when the data transfer setup completion frame“FCP_XFER_RDY” is received, the SC2022 acquires the reply responderExchange ID (RX_ID) received from the diskarray subset 10, specifies theS_ID, D_ID, OX_ID, instructs the SP2021 and registers the RX_ID in theapplicable Exchange of the ET2026. The SC2022 acquires the host I/F nodeNo. of the transfer responder (transfer originator of the command frame)for the data transfer completion frame. The SC2022 renders the S_ID ofthis frame invalid and transfers it to the SPG2024. The SPG2024generates the S Packet as described previously and transfers the SPacket to the corresponding host I/F node “#2” by way or the crossbarswitch 201.

When the SPG2024 in the host I/F node “#2” receives the S Packet of thedata transfer completion frame, the expansion header of the S Packet isremoved, and the “FCP_XFER_RDY” reproduced and delivered to the SC2022(step 20011). The SC2022 instructs the SC2021, searches the ET2026 andspecifies the applicable Exchange (step 20012). Next, the SC2022investigates whether the frame is “FCP_XFER_RDY” (step 20013) and if“FCP_XFER_RDY”, instructs the SP2021 to rewrite he originator ExchangeID (RX_ID) of ET2026 The value added to this frame is used as theoriginator Exchange ID (step 20014). The SC2022 then converts the S_ID,D_ID of the frame header 401 to an appropriate value used by the ID ofthe host 30 and the ID of the host I/F node 203 (step 20015) The frameheader 401 is thus converted to a frame corresponding to the host “#2”by means of this processing. The IC2023 issues a “FCP_XFER_RDY” datatransfer completion frame for this host “#2” (arrow (d) of FIG. 10)(step 20016).

The diskarray I/F controller 1011 for the diskarray subset “0” generatesa data frame “FCP_DATA” for performing data transfer, and transfers itto the diskarray switch 20 (arrow (e) of FIG. 10). A limit of a maximumdata length of 2 kilobytes for one frame is set to limit the datatransfer length of the frame payload. When this data length exceeded,data frames just equal to the required number are generated and issued.An identical SEQ_ID is assigned to all the data frames Except for thecase where a plurality of frames are generated for the same SEQ_ID (inother words SEQ_CNT changes), data frame issue is the same as for thedata transfer setup completion frame. The diskarray switch 20 implementsconversion of the frame header 401 for the data frame “FCP_DATA” justthe same as for the data transfer setup completion frame. However, anRX_ID has previously been established when transferring the data frameso that the processing of step 20014 for the data transfer setupcompletion frame is skipped. After conversion of the frame header 401,he diskarray switch 20 transfer the data frame to the host “#2” (arrow(f) of FIG. 10).

Next, the diskarray subset “#0” of the diskarray I/F controller 1011generates a status frame “FCP_RSP” perform the end status transfer andissued this frame to the diskarray switch 20 (arrow (g) of FIG. 10). Inthe diskarray switch 20, the expansion header is removed from the SPacket by the SPG2024 just the same as the processing for the datatransfer setup completion frame, the “FCP_RSP” frame is recreated (step20021) and the ET2026 is searched by the SP2021 and the Exchangeinformation acquired (step 20022). The SC2022 converts the frame basedon this information (step 20023). The converted frame is transferred tothe port “#2” by the IC2023 (arrow (h) of FIG. 10) (step 20024).Finally, the SP2021 deletes the exchange information from the ET2026(step 20025).

The read processing is thus performed from the diskarray. In the writeprocessing for the diskarray system 1, only the transfer direction ofthe data frame is reverse and the processing is otherwise the same asthe read processing.

The diskarray switch 20 as shown in FIG. 3 is provided with anintercluster I/F 2040 in the crossbar switch 201. In the systemstructure shown in FIG. 1, an intercluster I/F 2040 is not used. In thediskarray switch of this embodiment, other diskarray switches can bemutually connected as shown in FIG. 14, utilizing the intercluster I/F2040. In this embodiment, only a total of eight diskarray subsets 10 andhost 30 can be connected in a single diskarray switch 20 however aplurality of diskarray switches 20 can be mutually connected byutilizing the intercluster I/F 2040 and an increased number ofdiskarrays and hosts 10 can be connected. In the system shown in FIG. 14for example, four diskarray switches 20 are used to connect up to atotal of 32 units of the diskarray subset 10 and the hosts 30, and datacan be mutually transferred between these subsets and hosts. In thisway, the number of diskarray subsets and the number of hosts that can beconnected are increased according to the need for performance and diskcapacity in this embodiment. Also, the capacity, performance andexpandability of connection units can be drastically improved sinceconnections can be made between the host—diskarray system by utilizingthe necessary amount of host I/F transfer bandwidth.

In the embodiment as described above, even if the performance of onediskarray subset unit is limited by the internal bus and the internalMPU, mutual connections can be made between the host and the diskarraysubset by utilizing a plurality of the diskarray subsets, by means ofthe diskarray switch. In this way, high performance car be achieved as atotal diskarray system. Even if the performance of a diskarray subset isrelatively low, high performance can be attained by utilizing aplurality of diskarray subsets. Accordingly, low cost diskarray subsetscan be connected in just the required number to match the scale of thecomputer system, and a diskarray system can be constructed at a costappropriate to the desired scale. Further, when improvement inperformance of increasing the disk capacity is required, then thediskarray subsets can be added in just the required amount. Stillfurther, since a plurality of diskarray switches can be utilized toconnect an optional number of hosts and diskarray subsets, a drasticimprovement can be made in the capacity, the performance or the numberof units for connection, and a system with high expandability obtained.Even still further, reduced elements of a diskarray system itself of theconventional art can be utilized in this embodiment so that large salesoftware that was previously developed can be utilized without changes,thus reducing development costs and achieving a short developmentperiod.

(Second Embodiment)

FIG. 15 is a block diagram of the computer system of the secondembodiment of this invention. In this embodiment, the structure differsfrom the first embodiment in that, in the host I/F node of the diskarrayswitch, only the frame header 401 is converted, the frame payload 402 isnot operated and also in that the diskarray switch, the host I/F and thediskarray I/F are not duplexed (duplicated). The elements of thestructure are therefore not greatly different from the first embodimentand a detailed description of those similar sections is omitted.

In FIG. 15, the diskarray subsets 10 are comprised of a plurality oflogical units (LU) 110. Each LU110 is configured as an independent LU.The serial numbers assigned to the LUN in the LU110 inside the diskarraysubsets 10 generally start from 0 (zero). Therefore, when showing to ahost 30, consecutive LUN for all LU110 in the diskarray system 1, thenconverting the LUN field for the frame payload 402 is necessary, thesame as in the first embodiment. In this embodiment, the LUN of thediskarray subsets 10 are shown unchanged to the host 30, so conversionof the frame payload 402 is not necessary and the control of thediskarray switches is extremely simple.

In the diskarray switches of this embodiment, it is assumed that aspecified diskarray subset 10 can be accessed for each host I/F node203. When one host I/F 31 is used in this case, only the LU110 in onediskarray subset 10 an be accessed. When accessing LU110 in a pluralityof diskarray subsets 10 from one host unit is needed, then that host isconnected to a plurality of host I/F nodes 203. Further, when settingaccess of LU110 of one diskarray subset 10 from a plurality of host 30,then loop topology or fabric topology can be utilized in the same hostI/F node 203 to connect to the plurality of hosts 30. When configured inthis way, during access of one LU110 from one host 10, a diskarraysubset 10 can be set for each D_ID of the host I/F node 203 so that theLUN of each LU can be shown as is, to the host 30.

Since in this embodiment, the LU of each LU110 inside the diskarraysubsets 10 can be shown unchanged to the host 30 for the above relatedreasons, then conversion of the LUN is no longer required in thediskarray switch 20. Accordingly, when the diskarray switch 20 receivesa frame from the host 30, only the frame header 30 is converted the sameas in the first embodiment, and the frame payload 402 is transferredwithout conversion to the diskarray subset 10. In the operation of eachsection of this embodiment, excluding the fact that the conversion ofthe frame pay load 402 is not performed, the embodiment is the same asthe first embodiment so that a detailed explanation of the identicalsections is omitted The diskarray switch 2 can be easily developed inthis embodiment.

(Third Embodiment)

In the second embodiment, in the host I/F node of the diskarray switch,only the frame header 401 is converted, however in the third embodimentdescribed hereafter, frame conversion, including the frame header is notperformed. The computer system of this embodiment is configured the sameas the computer system in the first embodiment as shown in FIG. 1.

In the first and second embodiments, the internal structure of thediskarray system 1 such as the number of diskarray subsets 10 and theconfiguration of the LU110 are concealed from the host 30. The host 30therefore sees the entire diskarray system 1 as one storage device. Incontrast, in this embodiment, the diskarray subset 10 is revealed to thehost 30, and the host 30 directly uses the D_ID of the frame header asthe port ID for the diskarray subset. By this arrangement the diskarrayswitch can control frame transfer just by complying with the frameheader information, and the fabric of the fiber channel in theconventional art can be used instead of the diskarray switch 20 toachieve an equivalent switch device.

The diskarray system configuration manager (means) 70 communicates withthe communication controller 106 of the diskarray subset 10 as well asthe communication means 204 of the diskarray switch 20 and acquires orsets structural information of the diskarray subsets 10 and thediskarray switches 20.

The diskarray switches 20 have a structure basically the same as thediskarray switches of the first embodiment as shown in FIG. 3. However,in this embodiment, the frame header information for frames issued fromthe host 30 is used unchanged to control frame transfer so that theconversion function of the first and second embodiments, in which aframe header is achieved by a DCT2027, SC2022, SPG2024 of the diskarrayI/F node 202 and host I/F node 203 of the diskarray switch, is notnecessary. The crossbar switch 201 in the diskarray switch 20, performstransfer of fiber channel frames between the host I/F node 203, and thediskarray I/F node 202, according to the frame header information.

In this embodiment, to achieve total management of the diskarray systemstructure with the diskarray system configuration manager 70, adiskarray management table (hereafter this table is called DCT, isprovided in the diskarray system configuration manager 70. The DCTcomprising the diskarray system configuration manager 70 consists of agroup of two tables; a Diskarray System Configuration Table 20270 and aDiskarray Subset Configuration Table 202720-202723. The host-LU in thisembodiment are all comprise as one LU so that the “LU Type” in theHost-LU-Configuration table 20271 are all “ILU”, and the “CLU Class” andCLU Stripe Size” are not significant.

The administrator operates the management console 5, communicates withthe diskarray system configuration manager 70 and acquires informationsuch as the number of disk units, and disk capacity of the diskarraysubset 10, and performs setting of the LU110 of the diskarray subset 10and setting of the RAID level. Next, the administrator communicates withthe diskarray system configuration manager 70 from the managementconsole 5, controls the diskarray switch 20 and sets related informationamong the host 30 and the diskarray subsets 10. This operationestablishes the structure of the diskarray system 1 and allows LU1 to beseen as the administrator wishes, from the host 30. The diskarray systemconfiguration manager 70 saves the above setting information, verifiesthe configuration according operation by the administrator and performschanges in the structure (configuration).

In this embodiment, once the diskarray system 1 is configured, aplurality of diskarray systems 1 can be handled the same as onediskarray system and without making the administrator aware of thepresence of the diskarray switch 20. Further in this embodiment, thediskarray subsets 10 and the diskarray switches 20 can be operatedtogether by means of the same operating environment and confirming theirconfiguration (or structure) and making changes in the configuration isalso simple. Still further in this embodiment, when substituting thediskarray system of this embodiment with a diskarray system used in theconventional art, no changes are made in the host 30 settings, and thestructure of she diskarray system 1 can work with the diskarray systemstructure used up until then, and interchangeability can be maintained.

(Fourth Embodiment)

A fiber channel was used in the host I/F in the first through thirdembodiments described above. In the embodiment hereafter described, aninterface other than the fiber channel might also be used.

FIG. 16 is a block diagram of the IC (Interface Controller) 2023 insidethe host I/F node 203, when the host I/F is parallel SCSI. An SCSIprotocol controller (SPC) 20230 performs the protocol control of theparallel SCSI. A fiber channel protocol controller (FPC) 20233 performscontrol of the fiber channel. A protocol exchanging processor (PEP)20231 converts the protocol of the serial SCSI of the fiber channel andthe parallel SCSI. A buffer (BUF) 20232 temporarily stores the data ofthe protocol being converted.

The host 30 in this embodiment, issues a SCSI command to the diskarrayI/F node 203. In the case of a read command, the SPC20230 stores this inthe BUF 20232 and reports reception of the command by breaking into thePEP 20231. The PEP 20231 uses the command stored in the BUF20232, artconverts the command to FPC20233 and sends it to the FPC20233. When theFPC20233 receives this command, in convert the command into a frameconfiguration and delivers it to the SC2022. At this time, the ExchangeID, Sequence ID, Source ID and Destination ID are added to PEP 20231capable of the following processing. The remaining command processing isperformed the same as in the first embodiment. When the setup of data iscomplete, the data array subset 10 issues a data transfer setupcompletion frame, and after the data transfer ends correctly, implementsissue of a status frame. In the period from the diskarray subset 10 tothe IC2023, while the frame header 401 and the frame payload 402 arebeing converted as required, the transfer of each frame is performed.The FPC20233 of the IC2023 receives the data transfer setup completionframe, then receives the data and stores it in the BUF 20232 and if thetransfer has ended correctly, receives the status report, and breaksinto the PTP20231 to report that transfer of data is complete. When thePTP20231 receives the break-in (interruption), the SPC20230 starts upand instructs the start of data transfer to the host 30. The SPC20230transmits the data to the host 30, and after confirming normalcompletion, interrupts the PTP20231 to report the data transfer endedcorrectly.

A parallel SCSI was used as an example here of a host I/F other than afiber channel however other interfaces can be implemented such as forESCON in the same manner as a host I/F to the main frame. Host I/F nodescorresponding for instance, to the fiber channel, parallel SCSI andESCON can be provided as the host T/F node 203 of the diskarray switch20 so that all kinds of so-called open systems such a personal computersand work stations can be connected with the main frame to one diskarraysystem 1. In this embodiment, a fiber channel was utilized as thediskarray I/F in the first through the third embodiments however thedesired optional I/F can also be used as the diskarray I/F.

(Fifth Embodiment)

A method for configuration management of the diskarray system 1 isdescribed using the fifth embodiment. FIG. 17 is a system diagram ofthis embodiment. A total of four host 30 units are provided in thisembodiment. The I/F 30 connecting between the host “#0”, “#1” and thediskarray system 1 is a fiber channel, the host “#2” and the diskarraysystem 1 are connected by a parallel SCSI (Ultra SCSI). The host “#3”and the diskarray system 1 are connected by a parallel SCSI(Ultra2SCSI). The connection to the diskarray switch 20 of the parallelSCSI is performed in the same way as the fourth embodiment. Thediskarray system 1 has four diskarray subsets 30. The diskarray subset“#0” has four independent LU. The diskarray subset “#1” has twoindependent LU. The diskarray subset “#2” and the diskarray subset “#3”are comprised of one combined LU (CLU). In this embodiment, just thesame as the first embodiment, the diskarray subset 10 is concealed fromthe host 30, and the frame of the fiber channel is converted. The LUNassigned to each LU, in order from the diskarray subset “#0” are seven,LUN=0, 1, 2, . . . to 6.

FIG. 18 is a screen view showing on the management console screen 5.This figure shows the logical connection structure corresponding to thelogical units (LU) and the host I/F 31. The logical connectionconfiguration screen 50 shows the information 3100 relating to each hostI/F 31, the information 11000 relating to each LU110, and the relationof the diskarray subset 10 and the Lu110. Information relating to thehost I/F 31 includes the I/F type, the I/F speed and status, etc.Information relating to the Lu110 such as the storage subset No, LUN,capacity, RAID level, status, and information are displayed. Theadministrator refers to this information and can easily manage theconfiguration of the diskarray system 1. The lines drawn between thehost I/F and the LU on the logical connection configuration screen 50shows the LU110 accessible by way of each of the host I/F 31. ThoseLU110 to which a line is not drawn from the host I/F cannot be accessedfrom the host 30 connected to that host I/F. The data configuration thatis handled differs according to the host 30, and also differs accordingto the user so that appropriate restrictions on access must be providedin order to maintain security. The administrators setting the systemthereupon utilize this screen, to implement restrictions on access bygranting or denying access between the host I/F and each LU110. In thefigure, the LU “#0” can be accessed from the host I/F “#0” and “#1”however, the LU “#0” cannot be accessed from the host I/F “#2” and “#3”.The LU “#4” can only be accessed from the host I/F “#2”. In order toimplement these kind of access restrictions, the access restrictioninformation is sent from the diskarray system configuration manager 70to the diskarray switch 20. The access restriction information sent tothe diskarray switch 20 is distributed to each host I/F node 203 andregistered in the DCT2027 of each host I/F node 203. When an LU searchcheck command has been issued for an LU with access restrictions, thehost I/F node 203 performs a search of the DCT2027 and if a response isnot obtained to the search command or if an error is returned, than thatLU is no longer recognized (authorized) from the host. The Test UnitReady command of the Inquiry command are not typically used when in thecase of SCSI protocol as search command for the presence of an LU. Sinceread/write cannot be implemented without this search command,restrictions on access are easy to apply. In this embodiment, accessrestrictions are applied to each host I/F 31 however by extending thisthe implementing of access restrictions on each host 30 is easilyaccomplished. Further, the host I/F 31, host 30, or an address space canbe specified, and access restrictions can be applied according to thetype of command so that read only, write only, read and write permit,and read/write prohibit are enforced. In this case, the host I/F No, thehost ID, the address space or the restriction command are specified asthe access restriction information and the restriction set in the diskaccess switch 20.

Next, the addition of another diskarray subset 10 is described. Whenadding a new diskarray subset 10, the administrator connects thediskarray subset 10 to be added, to an empty I/F node 202 of thediskarray switch 20. The administrator next operates the managementconsole 5 and presses the “Show Latest Status” button 5001 displayed onthe logical connection configuration screen 50. A picture showing thediskarray subsets not yet set appears on the screen (not shown indrawing) in response to pressing the button 5001. When the picture forthis diskarray subset is selected, the setup screen for the diskarraysubsets then appears. The on this setup screen, the administratorexecutes the various settings for the newly added diskarray subset.Items set on this screen include the RAID level and the LUconfiguration. Next, on switching to the logical connectionconfiguration screen of FIG. 19, the new diskarray subset and the LUappear. From here on, the settings for restricting access for the hostI/F 31 are made, and the “Setup Execution” button 5002 is pressed,access restriction information, as well as diskarray subsets, and LUinformation for the diskarray switch 20 are transferred and the settingsenabled The procedure when adding a LU110 to the diskarray subset 10 isperformed the same as in the above related procedure. The deletion ofthe diskarray subset, and the LU are also performed with approximatelythe same procedure. One point of difference is that the administratorselects the sections for deletion on the screen and presses the “Delete”button, and the deletion is implemented after making an appropriatecheck. Thus by utilizing the management console 5, the administrator cancollectively manage the entire diskarray system.

(Sixth Embodiment)

Next the mirroring process by means of the diskarray switch 20 isdescribed utilizing the sixth embodiment The mirroring described here,is a method to support duplexed (duplicated) writing by means of twoindependent LU of two diskarray subsets, and duplicating including up tothe controller of the diskarray subset. The reliability therefore isdifferent from the method duplexing only the disks.

The system configuration, (structure) of this embodiment is the same asshown in FIG. 1. In the configuration of FIG. 1, the diskarray subsets“#0” and “#1” are provided with completely the same LU configuration.These two diskarray subsets are seen from the host 30 as one diskarray.For reasons of convenience, the pair No. of the diskarray subset thatwas mirrored is called “#01”. Also, a mirroring pair is formed by the LU“#1” and the LU “#0” of the diskarray subset, and this LU pair isconveniently named, LU “#01”. Information for managing the LU#01 is setas “Mirrored” in the CLU class on the Host-LU Configuration Table 20271of the DCT2027, and information relating to LU#0 and LU#1 is set as theLU Info. The configuration of the other sections is the same as in thefirst embodiment.

The operation of each section of this embodiment is largely the same asthe first embodiment. Hereafter, the points differing from the firstembodiment are explained mainly with the operation of the host I/F nodeof the diskarray switch 20. FIG. 19 is a model diagram showing thesequence of frames being transferred in the writ. operation of thisembodiment. FIGS. 20A through 20D are flowcharts showing the processingin the host I/F node 203 during the write operation.

In the write operation, the write command frame (FCP_CMD) issued by thehost 30 is received by the IC2023 (arrow (a) of FIG. 19) (step 21001)The write command frame received by the IC2023 is processed the same asin steps 20002-20005 in the write operation described for the firstembodiment (step 21002-21005). The SC2022 searches the DCT2027 using theSP2021 and verifies that there is a write access request to the LU “#01”of the mirrored diskarray subset “#01” (step 21006). The SC2022 makesduplicates of the command frame that was received in FB2025 (step21007). The SC2022 converts the command frame based on the structuralinformation set in the DCT2027, and makes separate command frames forboth the LU “#1” and the LU “#0” (step 21008). The LU “#0” is herecalled the master LU, and the LU “#1” the slave LU. The command framesare also called respectively the master command frame and the slavecommand frame. Both of these separate frames are stored in the exchangeinformation in ET2026, and a command frame issued for the diskarraysubset “#0” and the diskarray subset “#1” (arrows (b0) (b1) of FIG. 19)(step 21009).

The diskarray subsets “#0” and “#1” receive the command frames and therespective, independent, data transfer setup completion frames“FCP_XFER_RDY” are distributed to the diskarray switch 20″ (arrows (c0)(c1) of FIG. 19). In the diskarray switch 20, the data transfer setupcompletion frames transferred by the same processing as in seeps20011-20013 of thread operation in the first embodiment, are processedin the host I/F node 203 (step 21011-21013). At the stage that the datatransfer setup completion frames from each diskarray subsets arearranged (step 21014), the SC2022 converts the master data transfersetup completion frames (step 21015), and after frame conversion by theIC2023 sends the frame to the host 30 (arrow (d) of FIG. 19) (step21015).

After receiving the data transfer setup completion frame, the host 30sends the data frame (FCP_DATA) to the diskarray switch 20 (arrow (e) ofFIG. 19). When the data frame from the host 30 is received by the IC2023(step 21031), the read command frame and the write command frame areboth stored in the FB2025, and a CRC check and frame header analysis areperformed (steps 21032, 21033). The ET2026 is searched by the SP2021based on the frame header analysis results, and the Exchange informationis acquired (step 21034). The SP2022 makes duplicates the same as duringthe write command frame (step 21035). One copy is sent to the LU “#0” ofthe diskarray subset “#0” and the other is sent to the LU “#1” of thediskarray subset “#1” (arrow (f0) (f1) of FIG. 19) (step 21037).

The diskarray subsets “#0” and “#1” receive each of the data frames,respectively write these frames in the disk unit 104, and set the statusframe (FCP_RSP) to the diskarray switch 20. When the SP2022 receives thestatus frames from the respective diskarray subsets “#0” and “1#”, theirrespective expansion headers are removed from their status frames, theframe header restored and the exchange information acquired from theET2026 (step 21041, 21042). When the status frames from both thediskarray subsets “#0” and “#1” are arranged (step 21043), conversion ofthe master status frame from the LU “#0” is performed (step 21044) afterchecking that the status has completed correctly, and the slave statusframe is deleted (step 21045). Then, the IC2023 sends a command frame tothe host to report correct completion (arrow (h) of FIG. 19) (step21046). Finally, the SP2021 deletes the exchange information of ET2026(step 21047).

The write processing n the mirrored structure is thus completed The readprocessing for the mirrored LU “#01” differs only in the direction ofdata transfer, and is performed largely the same as the above describedwrite processing except that the issue of a read command to twodiskarray subsets is not necessary, and a command frame can be issuedjust to either diskarray subset. A command frame for instance can beissued mainly to the master LU however for high speed operation, methodssuch as alternate issue of command frames for both the master/slave LUwill prove effective in distributing the load.

In the above related processing, in steps 21014 and step 21043, a replyfrom the two diskarray subsets LU “#0” and “#1” is awaited, bothsynchronized with and he process then proceeds. With this kind ofcontrol, handling of errors is simple since the process proceeds afterverifying the success of the processing for both of the diskarraysubsets. On the other hand this kind of control has the drawbackperformance declines since the overall processing speed depends on whichof the replies is slower. To resolve this problem, in the diskarrayswitch, control such as by proceeding to the next process withoutwaiting for a reply from the diskarray subset or a “Asynchronous type”control that proceeds to the next process at the point where a replyfrom either one of the diskarray subsets is received are possible. Theframe sequence when this asynchronous type control is used is shown bythe dashed arrow lines in FIG. 19. In the frame sequence shown by thedashed arrow lines, the sending of the data transfer setup completeframe to the host performed in step 21016, is implemented after theprocessing in step 21009, without waiting for the data transfer setupcomplete frame from the diskarray subset 10 In this case, the datatransfer setup complete frame sent to the host, is generated by theSC222 of the diskarray switch 20 (dashed arrow line (d′)). The dataframe from the host 30 is transferred to the diskarray switch 20 at thetiming shown by the dashed arrow line (e′) In the diskarray switch 20,this data frame is temporarily stored in the F32025. The SC2022 makes areply after receiving the data transfer setup complete frame from thediskarray subset 10, and transfers the data frame held in the FB2025(dashed arrow lines (f0′), (f1′)) per the data transfer setup completeframe sent from the diskarray subset 10. The completion report to thehost 30 from the diskarray switch 20 is performed (dashed arrow line(h′)) when there is a report (dashed arrow lines (g0′), (g1′)) from bothof the diskarray subsets 10. This kind of processing can shorten theprocessing time by an amount equal to the time Ta shown in FIG. 19.

The following processing is implemented when an error occurs duringframe transfer between the diskarray subset 10 and the diskarray switch20. When the process being implemented is write processing, then a retryprocess is performed on the LU in which the error occurred. If the retryprocess is a success, then the process continues unchanged. However,when the retry process fails after a preset number of retries, then thediskarray switch 20 prohibits access to this diskarray set 10 (or LU)and information showing this prohibition is registered in the DCT2027.The diskarray switch 20 also reports this information to the diskarraysystem configuration manager 70 by way of the communication controller204 and the MP200. The diskarray system configuration manager 70 thenissues an alarm to the management console 5 in response to this report.The administrator can thus recognize that trouble has occurred.Afterwards, the diskarray switch 20 continues the operation by utilizinga normal diskarray subset. The host 30 also continues processing withoutrecognizing that an error has occurred.

This embodiment utilizes a mirror configuration in a two unit diskarraysubsystem to that the disk is made more resistant to problems that occurThe resistance of the diskarray controller, diskarray I/F, and thediskarray I/F node can also be improved, and the reliability of theoverall diskarray system can be improved without taking measures such asduplexing (duplicating) the internal buses.

(Seventh Embodiment)

In the seventh embodiment, a method is described for combining three ormore diskarray subsets 10 and configuring them into one logicaldiskarray subset group. In this embodiment, data is distributed andstored into a plurality of diskarray subsets 10. Distributing andstoring the data in this way allows distributing the access to thediskarray subsets, to prevent the access being concentrated inparticular diskarray subset so that the throughput of the total group isimproved. A diskarray switch is used in this embodiment to implementthis kind of striping.

An address map of the disk address system 1 of this embodiment is shownin FIG. 21. The address space for the diskarray subsets 10 is striped ata stripe size S. The address spaces of the disk address system 1 as seenfrom the host are distributed into the diskarray subsets “#0”, “1”, “#2”and “#3”. The size of the stripe size S is optional however should notbe reduced very much. If the stripe size S is too small, the possibilityof the occurrence of the stripe crossover, which is a phenomenon thatthe target data attaches to a plurality of stripes across diskarraysubsets, will be risen and overhead may occur in the process. When thestripe size S is set large, then the probability that stripe crossoverwill occurs diminishes, so a large stripe size S is preferable in termsof improved performance The a number of LU that can be set is optional.

Hereafter, the operation of the host I/F node 203 in this embodiment isdescribed while referring to the operation flowchart shown in FIG. 22and points differing from the first embodiment are described. In thisembodiment, as information relating to the striped Host-LU, “Striped” isset in the CLU Class and “S” is set in the CLU Stripe Size, in theHost-LU Configuration Table 20271 of the DCT2027.

When a command frame is issued from the host 30, the diskarray switch 20receives this command frame with the IC2023 of the host I/F node 203(step 22001). The SC2022 accepts this command frame from the IC2023,searches the DCT2027 using the SP2021 and verifies that striping isnecessary (step 22005). Next, SC2022 searches the DCT2027 using theSP2021, finds from the structural information containing the stripe sizeS, the stripe No. for the stripe belonging to the data being accessed,and designates what diskarray subset 10 this stripe is stored in (step22006). Stripe crossover may possible occur at this time however thisprocessing in such a case is related later. When no stripe crossoveroccurs, the SC2022 implements conversion of the command frame (step22007) based on SP2020 calculation results, and stores the exchangeinformation in the ET2026 (step 22008). The subsequent processing is thesame as for the first embodiment.

When stripe crossover has occurred, the SP2021 generates two commandframes. These frames are generated for instance, by duplicating thecommand frame issued from the host 30. New settings are made such as forthe frame header and frame payload of the generated command frame. Afterduplicating the command frame in SC2022, conversion can also beimplemented the same as in the sixth embodiment however in thisembodiment is newly made by SP2022. When the two command frames aremade, the SC2022 sends these frames to the respective diskarray subsets10. Data transfer is then performed the same as in the first embodiment.The point in this embodiment differing from the first embodiment is thatthe data itself must be transferred between one host 30 and twodiskarray subsets 10. In the read process for instance, the data frametransferred from the two diskarray subsets 10, must be transferred toall the hosts 30. The SC2022 at this time, complies with the informationregistered in the ET2026, and adds the appropriate exchange information,in the appropriate order to the data frame transferred from thediskarray subset 10 and sends this to the host 30. In the write process,two data frames are made, the same as for the command frame, andtransferred to the applicable diskarray subset 10. The sequentialcontrol of the data frames at the host or the diskarray subset is calledthe “Out of Order” function. This “Out of Order” function is notrequired if the configuration is compatible with nonsequentialprocessing. Finally, when all data transfer is complete, and thediskarray switch 20 has received the status frames respectively from thetwo diskarray subsets 10, the SP2021 (or the SC2022) make; a statusframe for the host 30, and the IC2023 sends this status frame to thehost 30.

This embodiment as described above, is capable distributing the access(load) into a plurality of diskarray subsets, so that along withimproving the total throughput, the access latency can be reduced

(Eighth Embodiment)

Next, the duplicating operation between the two, diskarray systems (orthe diskarray subsets) is described using the eighth embodiment. In thesystem described here, one of two diskarray systems is installed at aremote location to provide recovery assistance in case of damage to theother diskarray system due to a natural or man-made calamity, etc. Thiskind of countermeasure for dealing with damage from disasters isreferred to as disaster recovery and the making of copies performed withthe diskarray system at the remote location is referred to as remotecopy.

In the mirroring as described in the sixth embodiment, the mirrorfunction is achieved with the diskarray subsets 10 installed at largelythe same location geographically so that diskarray I/F 21 can use afiber channel. However when diskarrays (diskarray subsets) areperforming remote copy at remote locations in excess of 10 kilometers,then a fiber channel cannot be used to transfer a frame unless relayequipment is added. A mutual distance of some several hundred kilometersis used during disaster recovery so that use of fiber channels forconnecting between diskarrays is impractical. Therefore methods such assatellite communications or high speed public telephone lines with ATM(Asynchronous Transfer Mode) are utilized.

FIG. 23 is a block diagram of the disaster recovery system of theembodiment. In the figure, the reference numeral 81 denotes site A, 82denotes site B. Both sites are installed at geographically remotelocations. Reference numeral 9 denotes a public telephone line, throughwhich the ATM packet passes. The site A81 and the site B82 each have adiskarray system 1. In this case, the site A81 is the normally usedsite, while site B82 is used as the remote disaster recovery site whensite A81 is down due to a disaster. The contents of the diskarray subset“#0” and “#1” of the diskarray system 10 of the site A81 are copied tothe remote copy diskarray subset “#0” and “#1” of the diskarray system10 of site B82. The node for connection to the remote site from amongthe I/F nodes of the diskarray switch 20 is connected to the publictelephone line by utilizing ATM. This node is called the ATM node 205.The ATM node 205 is configured the same as the host I/F node shown inFIG. 5, and the IC2023 performs ATM—fiber channel conversion. Thisconversion is achieved by same method as the SCSI—fiber channelconversion in the fourth embodiment.

The remote copy process in this embodiment is similar to the mirroringprocess in the sixth embodiment. The points differing from the mirroringprocess of the sixth embodiment are explained next. When the host 30issues a write command frame., the diskarray system 10 of site A81performs frame duplicating the same as in the sixth embodiment, andtransfers one of the copied (duplexed) frames to its own diskarraysubset 10. The other frame is converted from a fiber channel frame to anATM packet by the ATM node 205 and sent to the site B82 by way of thepublic telephone line 9. At the site B82, the ATM node 205 of thediskarray switch 20 receives this packet. The IC2023 of the ATM node205, restores the fiber channel frame from the ATM packet, and transfersthe fiber channel frame to the SC2022 The SC2022 implements frameconversion the same as when the write command was received from the host30 and transfers the frame to the remote copy diskarray subset. Fromhereon, fiber channel—ATM conversion is performed for all the datatransfer setup completion frames, data frames and status frame, and byimplementing the same frame transfer process, remote copy can beachieved. When the read command frame was issued from the host 30, thediskarray switch 20 transfers the command frame only to the diskarraysubset 10 only for its own site and reads this data only from thediskarray subset 10 of its own site. The operation at this time is thesame as in the first embodiment.

This embodiment is capable of making backups of user data in real-timeand providing recovery assistance when damage has occurred to adiskarray system site due to a disaster, etc.

(Ninth Embodiment)

The combining of a plurality of LU in one diskarray subset 10 isdescribed next. The disk storage device for a main frame for instance,has a logical volume size set to a maximum value of 2 GB in order tomaintain interchangeability with the previous system. When using thiskind of diskarray system as an open system, the LU receive the samerestrictions on the logical volume size, so that the hosts see thisconfiguration as a large number of small size LU. This kind of methodhas the problem that operating the system is difficult when the systemhas developed to a high capacity level. To deal with this problem, amethod was contrived for combining these logical volume (in other wordsLU) units into one large combine LU (CLU) structure by means of thediskarray switch 20. The forming of a combined LU (CLU) is acheived inthis embodiment by the diskarray switch 20. The combining of LU in thisembodiment is the same as the forming of combined LU by means of aplurality of diskarray subsets 10 in the first embodiment. The differingpoint is only that in this embodiment, a plurality of LU are combinedwithin the same diskarray subset 10. The operation as a diskarray systemis completely the same as in the first embodiment.

By combining a plurality of LU in the same diskarray subset 10 in thisway, to form one large LU, a diskarray system is achieved havingexcellent operability, reduced management cost and in which there is noneed for the host to manage a large number of LU.

(Tenth Embodiment)

Next, a method far setting alternative paths by means of the diskarrayswitch 10 is explained while referring to FIG. 24. The structure of eachsection in the computer system shown in FIG. 24 is the same as in thefirst embodiment. Here, it is assumed that the two hosts 30 areaccessing the diskarray subset 10 by utilizing the different diskarrayI/F 21. The diskarray subsets, the host I/F nodes 203 of the diskarrayswitch 20 and the diskarray I/F nodes 202 in the figure are shown onlyin the numbers required for this explanation. The diskarray subset 10has the same structure as shown in FIG. 2, with two diskarray I/Fcontrollers each connected to one diskarray switch 20. An alternativepath for the diskarray I/F 21 is set in the DCT227 of each node of thediskarray switch 20. The alternative path is a substitute path toprovide access in the event trouble occurs on a particular path.

Here, the alternative path for the diskarray I/F “#0” is set as thediskarray I/F “#1”, while the alternative path for the diskarray I/F“#1” is set as the diskarray I/F “#0”. Alternative paths are set in thesame way respectively for the host adapter in the diskarray subset 10,the cache memory/shared memory, and the lower adapter.

Next, the setting of the alternative path is described, assuming that aproblem has occurred and the path connecting the diskarray I/F 21 to thehost adapter “#1” of the diskarray subset 1 is broken or unusable asshown in FIG. 24. At this time, the host “#1” utilizing the diskarrayI/F 21 where the problem occurred, is unable to access the diskarraysubset 10. The diskarray switch 20 detects an abnormality in the frametransfer with the diskarray subset 10 and when the path cannot berestored after retry processing is implemented, verifies a problem tohave occurred on this path. When a problem occurs on the path, theSP2021 registers the information that a problem has occurred in thediskarray I/F “#1” in the DCT2027. Hereafter, the SC2022 of the host I/Fnode 203 functions to transfer frames from the host “#1” to thediskarray I/F node 202 connected to the diskarray I/F node “#0”. Thehost adapter 101 of the diskarray subset 10 continues the processing ofthe command from the host “#1”. The diskarray switch 20 reports theoccurrence of a problem to the diskarray system configuration manager70, and the occurrence of a problem is then reported to theadministrator by means of the diskarray system configuration manager 70.

The embodiment described above, can therefore switch to an alternativepath when a problem occurs on a path, without this switch beingrecognized by the host and render the setting of substitutes on the hostside unnecessary. Thus the utilization of the system can be improved

In this invention as described above, a storage system can be achievedthat easily improves the storage device expandability, and reliabilityaccording to various requirements and the scale of the computer system.The above explanations of the each of the embodiments all utilized adiskarray system having a disk device. However, this, invention is notlimited to use of a disk device as a storage media and is alsoapplicable to optical disk devices, tape devices, DVD devices andsemiconductor storage devices, etc.

What is claimed is:
 1. Storage system comprising: a switch; and astorage sub-system coupled to said switch; and a management unit coupledto said switch and said storage sub-system for setting a configurationinformation of the storage system, wherein said switch includes; aplurality of first nodes each of which is coupled to a host; and aplurality of second nodes coupled to said storage sub-system, whereinsaid storage sub-system includes; a plurality of storage devices; aplurality of first ports each of which is coupled to one or a pluralityof storage devices; and a plurality of second ports each of which iscoupled to one of said second nodes, wherein each of said second portsis coupled to a plurality of said first ports.
 2. Storage systemaccording to claim 1, wherein the number of said first nodes is morethan the number of said second ports.
 3. Storage system of claim 1,wherein said management unit sends configuration information of storageareas in said storage sub-system to said switch.
 4. Storage system ofclaim 3, wherein said configuration information of storage areasincludes information of logical units in said storage sub-system. 5.Storage system of claim 1, further comprising: a memory coupled to saidplurality of first ports and said plurality of second ports.
 6. Storagesystem of claim 1, wherein each of said plurality of first portsreceives information required for accessing one of said plurality ofstorage devices from one of said plurality of second ports.
 7. Storagesystem of claim 1, wherein said management unit sets path informationbetween each of said plurality of hosts and said storage sub-system tosaid switch.
 8. Storage system of claim 1, wherein said management unitsets path information between each of said plurality of hosts and alogical unit in said storage sub-system to said switch.
 9. Storagesystem of claim 1, wherein said switch receives an access request fromone of said plurality of hosts, and sends said access request to saidstorage sub-system via one of said plurality of second nodes, and when acommunication error is occurred between said storage sub-system and saidone of said plurality of second nodes, said switch changes a pathbetween said storage sub-system and said switch and sends an accessrequest to said storage sub-system via another one of said plurality ofsecond nodes.
 10. Storage system of claim 1, wherein said switchrestricts access to a predetermined logical unit in said storagesub-system from a predetermined host.
 11. Storage system of claim 10,wherein said switch receives a command for accessing said predeterminedlogical unit from said predetermined host, and sends error response backto said predetermined host or sends no response back to saidpredetermined host.
 12. Storage system comprising: a plurality ofswitches each of which is coupled to a plurality of hosts; a pluralityof host adapters each of which is coupled to one of said plurality ofswitches, a plurality of disk adapters each of which is coupled to saidplurality of host adapters; a plurality of storage units each of whichis coupled to one or plurality of said plurality of disk adapters; and amanagement unit coupled to one or a plurality of said switches formanaging said storage system.
 13. Storage system according to claim 12,wherein each of said plurality of switches includes a plurality of firstnodes each of which is coupled to a host, and a plurality of secondnodes each of which is coupled to one of said host adapters, wherein thenumber of said first nodes is more than the number of said hostadapters.
 14. Storage system according to claim 13, wherein each of saidplurality of switches receives an access request from one of saidplurality of hosts via one of said plurality of first nodes and sendssaid access request to one of said plurality of host adapters via one ofsaid plurality of second nodes.
 15. Storage system according to claim14, wherein said access request has Destination ID Information. 16.Storage system according to claims 15, wherein said Destination IDInformation indicates said one of said plurality of first nodes, andeach of said plurality of switches sends request to said one of saidplurality of host adapters.
 17. Storage system of claim 13, wherein eachof said plurality of host adapters includes a plurality of ports each ofwhich is coupled to one of said plurality of second nodes.
 18. Storagesystem of claim 17, wherein each of said plurality of switches receivesan access request from one of said plurality of hosts, and sends saidaccess request to one of said plurality of ports in one of saidplurality of host adapters via one of said plurality of second nodes,and when a communication error is occurred between said one of saidplurality of ports and said one of said plurality of second nodes, eachof said plurality of switches sends an access request to another one ofsaid plurality of ports in said one of said plurality of host adaptersvia another one of said plurality of second nodes.
 19. Storage systemaccording to claim 15, wherein said Destination Id Information indicatesone of said plurality of host adapters, wherein each of said pluralityof switches sends said access request to said one of said plurality ofhost adapters.
 20. Storage system according to claim 19, wherein saidaccess request has a logical unit number and a logical block address.21. Storage system of claim 12, further comprising: a memory coupled tosaid plurality of host adapters and said plurality of disk adapters. 22.Storage system of claim 12, wherein each of said plurality of diskadapters receives information required for accessing one of saidplurality of storage units from one of said plurality of host adapters.23. Storage system of claim 12, wherein said management unit sets pathinformation between each of said plurality of hosts and a logical unitin said plurality of storage units to a switch coupled to saidmanagement unit.
 24. Storage system of claim 12, wherein each of saidplurality of switches restricts access to a predetermined host adapterfrom a predetermined host.
 25. Storage system of claim 12, wherein eachof said plurality of switches restricts access to a predeterminedlogical unit in said plurality of storage units from a predeterminedhost.
 26. Storage system of claim 12, wherein said management unit sendsconfiguration information of logical units included in said plurality ofstorage units to a switch coupled to said management unit.
 27. Storagesystem comprising: a switch coupled to a plurality of hosts; a pluralityof host adapters each of which is coupled to said switch; a plurality ofdisk adapters each of which is coupled to said plurality of hostadapters; and a plurality of storage units each of which is coupled toone or plurality of said plurality of disk adapters, wherein said switchcomprises a node via which said switch receives configurationinformation of storage areas of said plurality of storage units. 28.Storage system of claim 27, wherein said switch includes a plurality offirst nodes each of which is coupled to a host, and a plurality ofsecond nodes each of which is coupled to one of said plurality of hostadapters, wherein the number of said first nodes is more than the numberof said host adapters.
 29. Storage system comprising: a switch; and aplurality of storage sub-system each of which is coupled to said switch,wherein said switch includes; a plurality of first nodes each of whichis coupled to a host; and a plurality of second nodes each of which iscoupled to one of said storage sub-systems, wherein each of saidplurality of storage sub-systems includes; a plurality of storagedevices; a plurality of disk adapters each of which is coupled to one ora plurality of storage devices; and a plurality of host adapters each ofwhich is coupled to said plurality of disk adapters and one of saidplurality of second nodes, and wherein said switch comprises aninterface for receiving an access request including destinationinformation indicating one of said plurality of first nodes from a host,and sends the access request to one of said plurality of host adaptersvia one of said plurality of second nodes.